A few days ago while pondering relative kernels, I thought it might be useful to introduce reference kernels. A relative kernel would only be able to refer to a reference kernel. Which would require a slightly higher fee. That means there would be much fewer kernels to refer to, which would reduce resource usage of the necessary indexing data structures.

But then I realized such a distinction on kernel types is a privacy leak, so I scrapped the idea...

Anyway, thought I'd write it down for future reference:-)

Regarding

A "relative lock height" kernel would need an additional 32 bytes of data to store a reference

It only needs 8 bytes, namely the leaf or full index of the reference kernel in the kernel MMR.
When computing the kernel_sig_msg, this index is used to serialize the actual referenced kernel.
When verifying the kernel, this index is needed anyway to do the relative height check.

We could allow the broadcasted tx to have the full 32 byte reference kernel, and let the miners convert it to an 8 byte index. But normally you would only broadcast such a tx if you see the references kernel being confirmed, so it may be simpler to require the broadcaster to set the index. Then only the slate would have the 32 byte reference kernel.

It only needs 8 bytes, namely the leaf or full index of the reference kernel in the kernel MMR.

Yes. We have discussed this before I think? And then I forgot about it...
The slight complexity is who is responsible for converting indices to commitments, as you say.
Let me think about this a bit more and we can add this to the RFC.

But normally you would only broadcast such a tx if you see the references kernel being confirmed, so it may be simpler to require the broadcaster to set the index.

Agreed. I think this is the conclusion we came to previously. The parties transacting (via a slate) would require the full excess commitment. But once the tx is built and ready to broadcast we can translate these to the more compact MMR indices.

The kernel excess would be required when verifying the signature, but verifiers can quickly lookup the excess based via the provided MMR index.

The one edge-case here that we need to think through. If we do support a height of 0 for a relative lock height then we can potentially build a tx that references a kernel that does not yet exist in the MMR, it will be added in the same block as the new tx. In this situation we would not yet have a MMR index for the referenced kernel. Maybe we do not allow this and the relative height must be non-zero?

If we do allow relative height of 0 then we would potentially also need to handle the case where k1 references k2 and k2 itself has a circular reference back to k1. Maybe we do not want to go down the path of permitting this...

The big benefit of requiring relative height to be non-zero is the fact that a referenced kernel must have a lower MMR index than the kernel referencing it. This would not always be the case for two txs within the same block as no guarantees are made about the order of kernels within a block.

Indeed this was discussed before.
A downside is that kernel_sig_msg is no longer a function of just the kernel features, as it depends on the kernel MMR. We probably have to accept the ugliness of passing that as an argument to kernel_sig_msg...

The other complexity here is how to deal with forks and reorgs if we are passing the MMR index as part of the tx (and not the referenced kernel excess).
If block 100 puts kernel k1 at pos 100 but block 100' puts k1 at pos 101 then some nodes potentially see a subsequent relative height locked tx as invalid based on what appears to be an incorrect kernel index.

So we may need to reference kernels by the excess commitment up to the point where they are actually added to a block (by the miner).

Ah yes, you're right; replacement of the commitment by an index can only happen when the later kernel is added to a block (as the past is immutable from this block's viewpoint); i.e. by a miner.

Ah yes, you're right; replacement of the commitment by an index can only happen when the later kernel is added to a block (as the past is immutable from this block's viewpoint); i.e. by a miner.

Yes. I remember some of the earlier conversations now (one reason why we should be documenting these proposals as RFCs...) As long as txs are re-orderable (i.e. before inclusion in a block) we need to explicitly reference the excess commitment itself.

And we do need to be careful with handling reorgs and forks from the perspective of the txpool as a reorg can order previously accepted txs differently - txs with relative lock heights can re-enter the txpool and we need to revalidate the lock height criteria carefully.

In the top comment I wrote:

That means there would be much fewer kernels to refer to, which would reduce resource usage of the necessary indexing data structures.

After more careful thought, there is another resource we should be more concerned about.
That is the effort needed in the Initial Block Download. For Grin to be truly scalable, we should aim the IBD effort to be at most linear in UTXO set size, not in total number of transactions. So how can we hope to achieve that?
In my opinion, the most promising approach is through the use of proof techniques such as STARKs
(Scalable Transparent ARguments-of-Knowledge). The idea is to accompany the UTXO set with some proof that there exists some valid kernel history, consistent with all block headers, that validates said UTXO set. It is currently quite feasible to construct such proofs on a beefy machine for around a million kernels, and with future hardware and software improvements this is expected to go up by another 2 orders of magnitude. We could aim to build one STARK proof for every year of kernel history.

BUT this approach would be severely handicapped by kernels that can freely reference older kernels.
Performing an IBD with a STARK proof of kernel history, only provides knowledge of kernels beyond the STARK timeframe, which can be as short as the STATE_SYNC_THRESHOLD of two days.

It may therefore be wise to limit the relative height to that same horizon.
If we then want to break up the whole block chain history into shorter segments that can be handled by STARKs, then the segments would only need to overlap by this horizon, in order to validate all relative height constraints.

Actually, I find 2 days to be a little short, and would prefer to change STATE_SYNC_THRESHOLD to match CUT_THROUGH_HORIZON, i.e. one week.

Can anyone think of a use case of relative kernels where one week would be considered insufficient?

Ignoring specifics around STARKs for a second - it sounds like the issue could be restated to -

Do we want to allow arbitrary relative heights between relative kernels and the kernels they reference? And if we want to limit this to a shorter period what should the limit be?

Intuitively it makes sense to have some kind of limit and not simply allow arbitrary relative heights here.

Some follow up questions -

• Would it be easier to introduce a more restrictive limit at a later date?
  • Can we do this via a soft fork?
• Or would it be easier to start off with one that we then decide is overly restrictive - can we relax it at a later date?
  • Does this need to be a hard fork?

Do we want to allow arbitrary relative heights between relative kernels and the kernels they reference?

I would argue that we don't.

And if we want to limit this to a shorter period what should the limit be?

I think a week is quite reasonable. With payment channels, monitoring the chain for illicit close transactions is a minimal burden with a 1 week window. I suspect very few people would want to wait longer than a week to reclaim funds from a channel closure. Most will probably set it shorter.
Btw, a one week limite allows the relative height to fit in 16 bits...

Would it be easier to introduce a more restrictive limit at a later date? Can we do this via a soft fork?

Yes, a graph-rate majority of miners can enforce shorter limits, by considering longer limits invalid.

Or would it be easier to start off with one that we then decide is overly restrictive - can we relax it at a later date? Does this need to be a hard fork?

That is indeed a hard-fork, as all nodes would need to upgrade to accept the longer limits.

Just starting to pick this up again.
I still see no reason why a 1 week limit would not be sufficient - most would be significantly shorter than that.
And if it lets us represent the relative height in a u16 then even better.

A lot of the preparation work has been done in advance -

• we already support "variable sized" kernels (serialization protocol v2)

  • coinbase kernels are minimal
  • height locked kernels have a fee and an absolute lock height
  • plain kernels have only a fee
  • the proposed "relative" kernels will have a fee and lock height and reference to previous kernel

• these are supported in -

  • tx p2p msgs
  • block p2p msgs (and compact blocks)
  • local db storage (1 week of full blocks)
  • underlying txhashset (kernel MMR data file)

I think adding the new kernel feature variant will be relatively straightforward.

We would need to support this additional kernel functionality in the wallet slate. This may be where a lot of the outstanding effort will be needed.

If we decide to support both commitment and the more compact index representation (once previous kernel is in the MMR we know the index) we would need to implement the mechanism to look these up and to make kernel validation aware of the MMR (for the lookup to take place).

The 1 week limit would give us -

• a relatively small search space in which to look these up (we would only need to cache ~ 1 week of kernel data indices)
• but at a cost of some added complexity - txs could become invalid after being added to the txpool (this needs some thinking through)

On that 2nd point - a tx that was initially valid (within the 1 week limit) could still be in the txpool after the 1 week limit had passed, invalidating the tx. We would need some way to handle this reliably and robustly. They would effectively age out and need to be removed from txpool over time.
This would only be an issue once the txpool is larger than a single block, which is not the case currently.

To summarise previous discussions around "relative kernels" we have three different ways of representing the reference between one kernel and another earlier kernel -

• excess commitment
• MMR index
• Merkle proof (excess commitment at specific MMR index)

Each is beneficial under different conditions.

Transaction kernels are serialized as part of various larger structures -

• the tx slate
• a transaction p2p msg
• a block (and compact block) p2p msg

The excess commitment is included as part of the hash, so we must always know this. And hashing is pervasive and deeply embedded in the Grin code. There is no easy way to provide additional "extra data" when hashing various data structures.

When building a tx we may not necessarily know the MMR index so we can only use the excess commitment. So slates will need the excess commitment.

When broadcasting transactions and accepting them in the txpool we can assume the referenced kernel has already been appended to the kernel MMR and thus has a known MMR index. Recent kernels indices are subject to change though in a fork/reorg scenario.

We want the transaction acceptance decision to be fast when handling transactions at the txpool layer. So ideally transaction would be self-contained and would not require db lookups or reads from disk. So the MMR index here is not ideal as we would need to go look the referenced kernel up from the current kernel MMR.

At the block level the kernel MMR index is sufficient. MMR indices will be known and will be stable for the processing of a given block.

There is another possible complication with relying on kernel MMR indices. We currently only require nodes to maintain full kernel history to allow other nodes to fully sync on the network. There is no technical reason why a node could not process and validate all historical kernels and then discard them. Once validated they are no longer required.

If we assume some nodes do not maintain full kernel histories (to support this in the future) then we need to use Merkle proofs for relative kernels.
We may be able to use an MMR index for recent kernels but anything beyond the threshold would need a full Merkle proof.

So I'm leaning toward the following -

Relative kernels always include the excess commitment of the referenced kernel. We need the excess commitment to hash the kernel correctly and cannot rely on needing to lookup the external referenced kernel during the hashing operation.

At the slate level (during tx construction) we must specify an excess commitment for the referenced kernel for relative lock heights. We may not yet know the kernel MMR index.

During tx broadcast we provide an associated Merkle proof, defining the exact index in the kernel MMR and therefore allowing the relative lockheight to be directly verified.

Transaction kernels in blocks also require an associated Merkle proof for the same reasons.

We may be able to optimize this and omit the Merkle proof for recent referenced kernels if we assume all nodes will maintain a limited full history of recent kernels. So recent referenced kernels could be reference via an MMR index and older referenced kernels via a full Merkle proof. But this would be in addition to the actual excess commitment which would always be provided.

Edit: Let's ignore Merkle proofs for now.

Each relative kernel contains both -

• referenced excess commitment (needed for hashing)
• the kernel MMR index where we will find it (for fast lookups during validation)

During tx building we set the kernel MMR index to 0 and effectively ignore it.
When we come to broadcast the tx we must update the kernel MMR index with the real value based on current kernel MMR state (tx is invalid otherwise).

Merkle proofs will come in useful for lightweight nodes that do not maintain full kernel history - but these can opt-in in some way and left as a future enhancement.
Roughly: lighweight node receives tx or block, determines it cannot validate due to relative kernel and can request Merkle proofs as necessary (to fill in gaps).

Closing, superseded by #47.